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FOREWORD 


1 .  This  standard  defines  the  Government’s  requirements  and  expectations  for  contractor 
performance  in  defense  system  acquisitions  and  technology  developments. 

2.  This  new-issue  SMC  standard  comprises  the  text  of  The  Aerospace  Corporation  report 
number  TR-20 1 3-002 15,  entitled  Test  Requirements  for  Ground  Systems. 

3.  Beneficial  comments  (recommendations,  changes,  additions,  deletions,  etc.)  and  any 
pertinent  data  that  may  be  of  use  in  improving  this  standard  should  be  forwarded  to  the  following 
addressee  using  the  Standardization  Document  Improvement  Proposal  appearing  at  the  end  of  this 
document  or  by  letter: 


Division  Chief,  SMC/ENE 
SPACE  AND  MISSILE  SYSTEMS  CENTER 
Air  Force  Space  Command 
483  N.  Aviation  Blvd. 

El  Segundo,  CA  90245 

4.  This  standard  has  been  approved  for  use  on  all  Space  and  Missile  Systems 
Center/Air  Force  Program  Executive  Office  -  Space  development,  acquisition,  and 
sustainment  contracts. 


Mr.  David  Davis,  GG-15,  DAF 
SMC  Chief  Systems  Engineer 


-/i  f  L  d 


Mr.  Nick  Awwad,  GG-15,  DAF 
SMC/ENE 
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1.  Scope 


1.1  Purpose 

This  standard  establishes  the  requirements  for  functional,  performance,  and  environmental  testing  of 
ground  systems,  subsystems,  and  units  that  support  launch  and  operation  of  launch  vehicles,  upper- 
stage  vehicles,  and  space  vehicles.  This  standard  establishes  requirements  to  test  the  system  in  all 
configurations  and  conditions  of  mission  operations.  In  addition,  a  uniform  set  of  definitions  of 
related  terms  is  established. 

1.2  Application 

This  standard  is  applicable  to  the  procurement  of  ground  systems  hardware  and  software  for  space 
systems  as  a  compliance  document  for  the  establishment  of  a  minimum  set  of  test  requirements.  The 
standard  applies  to  organizations  developing  any  portion  of  the  ground  system. 

The  test  requirements  in  this  standard  are  applicable  to  the  design,  development,  and  integration  of 
new,  reused,  or  modified  software  and  hardware  in  ground  systems  that  provide  command  and 
control,  mission  management,  mission  data  processing,  and  common  services  (e.g.,  data  storage, 
networks,  and  other  shared  functions)  for  space  vehicles.  They  also  apply  to  training  systems  and 
simulators  used  for  training  operators  and  maintainers  of  ground  systems. 

This  standard  applies  to  systems  installed  in  fixed  and  non-fixed  environments  that  directly  support 
launch  and  operation  of  space  systems.  This  standard  does  not  apply  to  ground  support  equipment 
used  for  transport,  handling,  and  test  support  of  space  and  launch  vehicles.  , 

The  term,  ground  system,  is  used  in  this  standard  for  the  top  level  item  to  be  tested.  Terminology  used 
in  a  specific  development  will  likely  differ,  particularly  large  or  complex  procurements.  For  example, 
large  space  program  procurements  frequently  use  the  “system”  designation  for  all  components 
developed,  to  include  a  space  segment,  a  ground  segment,  a  launch  segment,  and  a  user  segment. 
There  also  may  be  further  subdivision  of  a  ground  segment  such  as  two  or  more  ground  elements. 

The  definition  for  “Test  Item  Levels”  in  section  3  lists  the  test  levels  used  in  this  standard,  which 
should  be  translated  and  tailored  for  the  levels  and  terms  of  a  specific  program. 

The  test  requirements  herein  focus  on  design  verification  and  the  elimination  of  latent  defects,  not 
evaluation  of  workmanship  issues,  to  help  ensure  a  high  level  of  confidence  in  achieving  successful 
space  missions. 

1.3  Test  Categories 

The  test  requirements  discussed  herein  are  categorized  as  follows: 

1.3.1  Developmental  Tests 

Developmental  tests  are  planned  and  performed  by  the  developers  to  verify  correct  implementation 
and  compliance  with  specifications,  provide  evidence  for  acceptance  of  developed  systems,  and 
collect  data  to  certify  readiness  for  operational  testing.  Types  of  developmental  tests  include: 
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1.  Development  tests.  Tests  conducted  during  development  of  hardware  and  software  items  to 
confirm  design  adequacy  and  verify  that  hardware  manufacture  and  assembly,  and  software 
implementation  perform  as  designed. 

2.  Qualification  tests.  Tests  performed  to  demonstrate  that  hardware,  software,  or  integrated 
items  meet  specified  requirements. 

3.  Acceptance  tests.  Formal  tests  required  for  contractual  acceptance  of  a  ground  system. 

4.  Integrated  system  tests.  Tests  performed  with  the  new  ground  system  integrated  with  other 
program  elements  or  segments  under  development,  and  with  existing  installed  systems,  to 
verify  correct  interface  implementation  and  correct  end-to-end  operation  of  the  integrated 
systems. 

1.3.2  Operational  Tests 

Operational  tests  are  conducted  by  the  procuring  activity  or  a  designated  test  organization,  in 
coordination  with  the  operations  organization.  Test  support  is  provided  by  developers  as  designated  in 
the  statement  of  work  (SOW).  These  tests  are  conducted  to  test  and  evaluate  the  capability  of  the 
ground  system  to  meet  the  operational  requirements.  Operational  tests  may  be  combined  with 
developmental  tests  when  objectives  can  be  satisfied  concurrently. 

1.4  Test  Requirements 

Environments  other  than  those  specified  in  section  5. 1 . 1 .2  of  this  standard  can  be  sufficiently  stressful 
as  to  warrant  special  analysis  and  testing.  These  include  environments  such  as  combat  conditions,  or 
nuclear  and  electromagnetic  radiation. 
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2.  Applicable  Documents 


2.1  General 

The  documents  listed  in  this  section  are  specified  as  compliance  documents  in  Sections  3,  4,  or  5  of 
this  standard. 

2.2  Government  Documents 

2.2.1  Specifications,  Standards,  and  Handbooks 

The  following  specifications,  standards,  and  handbooks  form  a  part  of  this  document  to  the  extent 
specified  herein.  The  revisions  of  these  documents  to  be  used  are  those  specified  in  the  solicitation  or 
contract.  If  revision  is  not  specified  by  contract  or  other  agreement,  the  version  current  at  initiation  at 
initiation  of  the  development  is  to  be  used. 

DEPARTMENT  OF  DEFENSE  STANDARDS 


[1] 

MIL-STD-461 

Requirements  for  the  Control  of  Electromagnetic  Interference 
Characteristics  of  Subsystems  and  Equipment 

[2] 

MIL-STD-810 

Environmental  Engineering  Considerations  and  Laboratory 
Tests 

[3] 

MIL-STD-1 88-125-1 

Fligh-Altitude  Electromagnetic  Pulse  (HEMP)  Protection  for 
Ground-Based  C41  Facilities  Performing  Critical,  Time- 
Urgent  Missions,  Part  1,  Fixed  Facilities 

[4] 

MIL-STD-1 88-125-2 

High-Altitude  Electromagnetic  Pulse  (HEMP)  Protection  for 
Ground-Based  C41  Facilities  Performing  Critical,  Time- 
Urgent  Missions,  Part  2,  Transportable  Systems 

2.2.2  Other  Government  Documents,  Drawings,  and  Publications 

The  following  other  government  documents,  drawings,  and  publications  form  a  part  of  this  document 
to  the  extent  specified  herein.  Unless  otherwise  specified,  the  revision  of  these  documents  is  that 
specified  in  the  solicitation  or  contract.  If  revision  is  not  specified  by  contract  or  other  agreement,  the 
version  current  at  initiation  of  the  development  is  to  be  used. 

CODE  OF  FEDERAL  REGULATIONS 


[5] 

Federal  Code  of  Regulations 

Federal  Code  of  Regulations,  Title  47: 

FCC  Part  15 

Telecommunication,  Part  15-Radio  Frequency 

Devices 

2.3  Non-government  Publications 

The  following  documents  form  a  part  of  this  document  to  the  extent  specified  herein.  Unless 
otherwise  specified,  the  revisions  of  these  documents  are  those  cited  in  the  solicitation  or  contract.  If 
revision  is  not  specified  by  contract  or  other  agreement,  the  version  current  at  initiation  of  the 
development  is  to  be  used. 
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THE  AEROSPACE  CORPORATION 


[6] 


TOR-2004(3909)-3537B 


Software  Development  Standard  for  Space  Systems, 

March  11,  2005  [Issued  also  as  SMC-S-012  (2008),  Software 
Development  for  Space  Systems] 


INSTITUTE  OF  ELECTRICAL  AND  ELECTRONIC  ENGINEERS  (IEEE) 


[7] 

IEEE  Standard  C95 

IEEE  Standard  for  Safety  Levels  with  Respect  to  Human 
Exposure  to  Radio  Frequency  Electromagnetic  Fields,  3kHz  to 
300  GHz 

[8] 

IEEE  Standard  1100 

Recommended  Practices  for  Powering  and  Grounding 

Electronic  Equipment 

2.4  Order  of  Precedence 

In  the  event  of  a  conflict  between  the  text  of  this  document  and  the  references  cited  herein, 
the  text  of  this  document  takes  precedence.  Nothing  in  this  document,  however,  supersedes 
applicable  laws  and  regulations  unless  a  specific  exemption  has  been  obtained. 
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3.  Definitions 


Acceptance:  The  action  taken  by  the  acquirer’s  designated  representative  by  which  the  acquirer 
assumes  ownership  of  products  that  signify  partial  or  complete  performance  of  a  contract. 

Acceptance  test:  The  required  formal  tests  conducted  to  demonstrate  acceptability  of  an  item  for 
delivery.  The  tests  are  designed  to  demonstrate  performance  to  specified  requirements  and  to  act  as 
quality  control  screens  to  detect  deficiencies  of  workmanship,  material,  and  quality. 

Analysis,  verification  by:  Verification  of  a  specification  requirement  by  use  of  mathematical  or 
statistical  methods,  modeling  and  simulation  using  data  from  software  and  hardware  engineering 
documents  produced  during  product  development  or  provided  by  COTS  vendors,  from  prototypes,  or 
from  lower-level  tests.  Note:  Analysis  that  must  be  performed  on  the  data  collected  during  the 
demonstration  or  test  methods  is  an  integral  part  of  those  methods  and  should  not  be  confused  with 
the  analysis  method  described  here. 

Bug  (software):  Common  and  traditional  slang  for  a  defect  in  software  syntax  or  logic  that  causes 
anomalous  behavior  of  the  system. 

Build  (software):  Can  have  two  meanings:  (1)  the  version  of  software  that  meets  a  subset  of  the 
requirements  allocated  to  the  completed  software  system,  or  (2)  the  period  of  time  during  which  the 
version  with  the  subset  of  requirements  is  developed. 

Commercial  Off  the  Shelf  (COTS):  Commercial  items  that  require  no  unique  government 
modifications  or  maintenance  over  the  lifecycle  of  the  product  to  meet  the  needs  of  the  procuring 
agency.  A  COTS  item  (hardware,  software,  or  both)  is  produced  and  made  commercially  available  or 
placed  in  stock  by  a  vendor  prior  to  the  vendor  receiving  orders  or  contracts  for  sale  of  the  item.  The 
vendor  may  produce  the  item  to  either  commercial,  military,  or  federal  specifications  or  descriptions. 
COTS  includes  items  stocked  by  distributors  for  which  government  contracts  may  be  received. 

Component:  A  generic  designation  for  a  part  of  the  system  (not  used  to  describe  a  level  of  software 
hierarchy,  since  that  use  is  obsolete,  nor  as  a  synonym  for  a  hardware  part). 

Debug:  To  detect,  locate,  and  correct  faults  in  a  computer  program. 

Defect:  A  flaw  in  a  system,  hardware,  or  software  item.  The  cause  of  a  failure.  That  which  is 
corrected  to  correct  a  failure. 

Demonstration,  verification  by:  Verification  of  a  specification  requirement  by  observing  functional 
operation  of  the  system  or  part  of  the  system  without  the  use  of  instrumentation  or  special  test 
equipment  beyond  that  inherently  provided  in  the  system  being  verified. 

Department  of  Defense  (POD)  standard:  A  standard  used  to  satisfy  primarily  multiple,  military- 
unique  applications.  There  are  five  types  of  DOD  standards:  interface,  design  criteria,  manufacturing 
process,  standard  practices,  and  test  method  standards. 
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Dependability:  An  umbrella  term  for  six  system  attributes: 1 

•  Availability:  readiness  for  correct  service 

•  Reliability:  continuity  of  correct  service 

•  Safety:  absence  of  catastrophic  consequences  on  the  user(s)  and  the  environment 

•  Integrity:  absence  of  improper  system  alterations 

•  Maintainability:  ability  to  undergo  modifications  and  repairs 

•  Confidentiality:  absence  of  unauthorized  disclosure  of  information 

Developer:  The  groups  or  organizations  performing  the  development  of  the  ground  system,  including 
prime  contractor,  subcontractors,  team  members,  government  development  staff,  or  associated 
agencies’  development  staff.  A  team  member  is  any  internal  or  external  organization  that  develops, 
tests,  or  supports  work  for  the  ground  system  by  an  agreement  (formal  or  informal)  with  any  other 
developer. 

Discrepancy  (test):  A  functional  or  structural  anomaly  or  failure  which  indicates  a  possible  deviation 
from  specification  requirements  for  the  test  item.  A  test  discrepancy  may  be  a  momentary, 
nonrepeatable,  or  permanent  failure  to  respond  in  the  predicted  manner  to  a  specified  combination  of 
test  environment  and  functional  test  stimuli.  Test  discrepancies  may  be  due  to  a  failure  of  the  test  item 
or  to  some  other  cause,  such  as  the  test  setup,  test  instrumentation,  supplied  power,  or  test  procedures. 

Discrepancy  Report  (DR):  Description  of  the  discrepant  behavior  and  effect  on  the  system  component 
being  tested.  Report  may  also  include  information  for  managing  and  analyzing  defects  and  testing, 
such  as  proposed  correction,  due  date,  assignee,  completion  date,  source  (requirements,  design, 
implementation,  etc.)  or  cause  of  the  discrepancy.  Also  called  problem  report,  deficiency  report,  and 
other  similar  terms. 

Failure:  The  inability  of  a  system  or  component  to  perform  its  functions  as  specified  or  within 
specified  performance  parameters.  Behavior  of  a  system  or  system  component  that  deviates  from 
specified  requirements.  A  test  step  not  passed. 

Fault:  See  Defect. 

Firmware:  Firmware  is  a  combination  of  a  hardware  device  and  computer  instructions  or  computer 
data  that  reside  as  read-only  software  on  the  hardware  device.  The  software  cannot  be  readily 
modified  under  program  control. 

Functional  Configuration  Audit  (FCA):  Verifies  that  all  item  or  subsystem  requirements  established 
in  the  functional  and  allocated  baselines,  specifications,  and  test  plans  have  been  verified 
successfully,  and  corrective  action  has  been  initiated,  as  necessary. 

Inspection,  verification  by:  Verification  of  a  specification  requirement  by  examination  of  the  product 
using  the  human  senses  or  by  simple,  nonprecision  measurements  or  examination  of  software  and 
hardware  engineering  documents  produced  during  product  development  or  provided  by  COTS 
vendors. 


1  A.  Avizienis,  J.C.  Laprie,  B.  Randell,  and  C.  Landwehr,  "Basic  Concepts  and  Taxonomy  of  Dependable  and  Secure 
Computing,"  IEEE  Transactions  on  Dependable  and  Secure  Computing,  Vol.  1,  No.  1,  pp.  11-33,  January-March  2004. 
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Integration:  Connection  and/or  interoperation  of  newly  developed  or  modified  software  or  hardware 
test  items  into  configurations  that  allow  test  of  functions  and  operations  of  the  combined  items.  Also 
connection  and/or  interoperation  of  the  ground  system  with  other  items,  existing  or  in  development, 
e.g.,  spacecraft  or  other  ground  systems. 

Non-developmental  Item  (NDI):  An  NDI  is  any  previously  developed  item  of  supply  used  exclusively 
for  government  purposes  by  a  federal  agency,  a  state  or  local  government,  or  a  foreign  government 
with  which  the  United  States  has  a  mutual  defense  cooperation  agreement;  any  item  described  above 
that  requires  only  minor  modifications  or  modifications  of  the  type  customarily  available  in  the 
commercial  marketplace  in  order  to  meet  the  requirements  of  the  procuring  department  or  agency. 

Non-fixed  Equipment:  Equipment  that  is  transportable,  mobile,  or  portable.  Transportable  equipment 
is  deployable  and  redeployable  but  not  self-propelled,  requiring  separate  vehicle(s)  for  transport. 
Mobile  equipment  is  self-propelled  and  capable  of  mission  operations  while  moving  or  shortly  after 
stopping.  Portable  equipment  is  capable  of  being  carried  by  one  or  more  persons. 

Part:  A  part  is  a  single  piece,  or  two  or  more  pieces  joined  together,  which  are  not  normally  subject  to 
disassembly  without  destruction  or  impairment  of  the  design  use.  Some  examples  are  resistors, 
transistors,  integrated  circuits,  relays,  capacitors,  gears,  screws,  and  mounting  brackets. 

Physical  Configuration  Audit  (PCA):  Physical  examination  of  the  actual  configuration  of  the  item 
produced.  It  verifies  that  the  design  and  product  documentation  specified  in  the  contract  matches  the 
as-built  item  and  that  all  documentation/deliverables  to  the  operational  site  are  present,  complete,  and 
accurate. 

Procuring  Activity:  The  procuring  activity  is  the  government  office  or  agency  with  primary 
responsibility  for  developing  and  acquiring  the  system,  subsystem,  equipment,  computer  software,  or 
engineering  services  addressed  in  this  document. 

Problem  Report:  See  Discrepancy  Report. 

Regression  Test:  A  test  performed  after  system  changes  have  been  made  to  verify  that  the  changes  did 
not  inadvertently  introduce  failures  to  system  functions  that  were  working  properly  prior  to  the 
changes. 

Requirements  Verification  Traceability  Matrix  (RVTM):  See  Verification  Cross-reference  Matrix 
(VCRM). 

Software  Development  Environment:  The  computers  and  software  development  tools  used  to  conduct 
coding  and  unit  testing  of  a  software  item.  The  computers  and  support  software  may  not  be  identical 
to  the  target  hardware  and  support  software.  See  Software  Integration  and  Test  Environment. 

Software  Development  File  (SDF):  A  repository  for  material  pertinent  to  the  development  of  a 
particular  body  of  software.  Contents  typically  include  (either  directly  or  by  reference) 
considerations,  rationale,  and  constraints  related  to  requirements  analysis,  design,  and 
implementation;  developer  internal  test  information;  and  schedules  and  status  information. 

Software  Integration  and  Test  Environment:  The  developer-controlled  and  developer-maintained 
configuration  of  computer  and  communications  hardware,  test  and  measurement  software,  and 
equipment  used  to  perform  integration  and  qualification  testing  of  software. 
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Software  Item:  An  aggregation  of  software  that  satisfies  an  end-use  function  and  is  designated  for 
purposes  of  specification,  interfacing,  qualification  testing,  configuration  management,  or  other 
purposes.  A  software  item  comprises  one  or  more  software  units.  A  software  item  was  sometimes 
previously  called  a  computer  software  configuration  item.  See  also  Software  Subsystem. 

Specification:  A  document  used  in  development  and  procurement  that  describes  the  technical 
requirements  for  items,  materials,  and  services.  Specifications  may  be  unique  to  a  specific  program 
(program  peculiar)  or  they  may  be  common  to  several  applications  (general  in  nature). 

Subassembly:  The  term  subassembly  denotes  two  or  more  parts  joined  together  to  form  a  stockable 
unit  which  is  capable  of  disassembly  or  part  replacement.  Examples  are  a  printed  circuit  board  with 
parts  mounted,  or  a  gear  train. 

Subsystem,  Hardware:  A  subsystem  is  an  assembly  of  two  or  more  components,  including  the 
supporting  structure  to  which  they  are  mounted,  and  any  interconnecting  cables  or  tubing.  A 
subsystem  is  composed  of  functionally  related  components  that  perform  one  or  more  prescribed 
functions. 

Subsystem,  Software:  Synonymous  with,  or  instead  of,  software  item  in  some  programs.  The 
specification  level  consisting  of  software  requirements. 

System  of  Systems:  A  set  or  arrangement  of  interdependent  systems  that  are  related  or  connected  to 
provide  a  given  capability.  The  loss  of  any  part  of  the  system  will  significantly  degrade  the 
performance  or  capabilities  of  the  whole. 

System  Verification  Review  (SYR):  A  multidisciplinary  technical  review  to  ensure  the  system  is 
ready  to  proceed  into  low-rate  initial  production  and  full-rate  production  within  cost  (program 
budget),  schedule  (program  schedule),  risk,  and  other  system  constraints.  Generally  this  review 
provides  an  audit  trail  from  the  critical  design  review. 

Test:  Activities  conducted  to  obtain  data  to  verify  that  an  implementation  is  as  designed,  that 
specification  requirements  are  satisfied,  and  to  identify  defects  and  deficiencies  for  corrective  action. 
Tests  that  verify  these  requirements  can  use  verification  methods  described  in  the  definitions  for: 
Analysis,  verification  by;  Demonstration,  analysis  by;  Inspection,  analysis  by;  and  Test,  verification 
by.  The  narrower  use  of  the  term  ‘test’  is  one  method  to  verify  a  requirement,  as  described  in  the 
definition  for  Test,  Verification  by. 

Test  Documentation  File:  A  repository  for  material  pertinent  to  the  test  of  a  unit,  subsystem,  or  other 
component  of  a  ground  system.  Contents  typically  include  (either  directly  or  by  reference)  test  plans, 
procedures,  data,  results,  reports,  and  schedules  and  status  information.  Would  also  include  analysis 
of  anomalies,  and  root  cause  of  anomalies,  rationale  for  retest,  and  regression  test. 

Test  Environment:  Test  environments  for  hardware  consist  of  the  facility  housing  the  equipment 
under  test,  necessary  fixtures  or  special  handling  equipment,  plus  the  required  test  instrumentation. 
Test  environments  for  software  consist  of  the  computing  hardware  running  the  software,  test  drivers, 
databases  and  simulators,  data  collection  and  analysis  software,  test  execution  scripts,  or  automated 
test  software. 


Test  Item  Levels:  The  levels  used  in  this  document,  from  the  simplest  to  the  most  complex,  are: 


Hardware 


Software 


Part 

Subassembly 
Unit 

Subsystem 
System 

Additional  levels  such  as  “segment”  and  “element”  are  used  in  large  ground  “system-of-systems 
configurations.”  These  levels  will  be  defined  by  the  procuring  activity. 


Unit 

Software  Item 

Subsystem  (alternate  for  software  item  or  not  used  in  some  systems) 
System 


Test  Method  Standard:  A  standard  that  specifies  procedures  or  criteria  for  measuring,  identifying,  or 
evaluating  qualities,  characteristics,  performance,  and  properties  of  a  product  or  process. 

Test  Readiness  Review  (TRR):  A  multidisciplinary  technical  review  and  process  assessment  to  ensure 
that  a  subsystem  or  system  is  ready  to  proceed  into  formal  test.  The  TRR  assesses  test  objectives,  test 
methods  and  procedures,  scope  of  tests,  and  safety,  and  confirms  that  required  test  resources  have 
been  properly  identified  and  coordinated  to  support  planned  tests. 

Test,  verification  by:  Verification  of  a  specification  requirement  by  exercising  or  operating  the  system 
or  a  part  of  the  system  and  performing  measurements  using  hardware  or  software  instrumentation  or 
special  test  equipment  that  is  not  a  part  of  the  system  being  verified. 

Unit,  Hardware:  A  separately  testable  component  of  a  hardware  design  that  is  part  of  a  subsystem. 

Unit,  Software:  An  element  in  the  design  of  a  software  item;  for  example,  a  major  subdivision  of  a 
software  item,  a  component  of  that  subdivision,  a  class,  object,  module,  function,  routine  or  database. 
Software  units  in  the  design  may  or  may  not  have  a  one-to-one  relationship  with  the  code  and  data 
entities  (routines,  procedures,  databases,  data  files,  etc.)  that  implement  them  or  with  the  computer 
files  containing  those  entities.  A  software  unit  was  sometimes  previously  called  a  computer  software 
unit. 


Validation:  Confirmation  that  the  product  or  service,  as  provided  (or  as  will  be  provided),  will  fulfill 
its  intended  use.  In  other  words,  validation  ensures  that  “you  built  the  right  thing.”2 

Verification:  Confirmation  that  work  products  properly  reflect  the  requirements  specified  for  them. 
In  other  words,  verification  ensures  that  “you  built  it  right.”2 

Verification  Cross-reference  Matrix  (VCRM):  Section  of  each  specification  or  a  separate  document 
that  lists,  for  each  requirement,  the  requirement  identifier  and  the  verification  method  for  the 
requirement — inspection,  analysis,  demonstration,  or  test.  Called  Requirements  Verification 
Traceability  Matrix  (RVTM)  on  some  programs. 


2M.B.  Chrissis,  M.  Konrad,  and  S.  Shrum,  CMM1®  for  Development,  Guidelines  for  Process  Integration  and 
Product  Improvement,  Addison- Wesley,  NJ,  2011,  p.  603. 
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4.  General  Requirements 


4.1  Test  Requirements 

This  standard  establishes  a  minimum  set  of  test  requirements  to  be  included  as  part  of  a  complete  test 
program.  The  test  strategy  is  a  process  of  development,  integration,  and  qualification  test  at  unit, 
subsystem,  and  system  level  to  provide  the  highest  probability  of  meeting  specification  requirements 
reliably  under  all  operational  conditions.  The  specific  test  requirements  applicable  at  each  level  of  test 
are  presented  in  Section  5. 

4.2  Testing  Strategy 

The  complete  test  program  for  ground  systems  consists  of  developmental  tests  and  operational  tests. 

A  satisfactory  test  program  requires  the  completion  of  specific  test  objectives  in  a  specified  sequence. 
Ground  systems  that  support  space  and  launch  vehicles  are  usually  developed  iteratively  in  a 
sequence  of  developments  (e.g.,  builds,  spirals,  increments),  each  with  defined  tests.  In  many  cases, 
ground  system  development  activities  consist  of  modifications  and  integration  of  updates  into  existing 
ground  sites  and  systems.  As  such,  the  testing  strategy  may  be  streamlined  with  a  goal  toward 
efficiency;  however,  the  strategy  still  needs  to  ensure  thoroughness  of  the  testing  it  prescribes  and  be 
acceptable  to  the  procuring  activity.  The  overall  test  program  encompasses  the  testing  of 
progressively  more  integrated  ground  system  hardware  and  software.  Small  systems  may  not  use  an 
iterative  development  strategy  and  would  be  tested  as  a  complete  system.  The  phases  of  executing 
this  test  strategy  are  illustrated  in  Figure  1. 

Design  adequacy  and  correct  implementation  should  be  demonstrated  in  development  tests  prior  to 
formal  qualification  testing.  Test  plans  for  requirements  verification  follow  a  bottom  up  test 
philosophy  illustrated  in  Figure  2.  That  is,  the  requirements  and  hardware/software  functions  are 
verified  at  the  lowest  level  where  test  configuration  and  environmental  conditions  constitute  a  valid 
test. 

Tests  must  be  performed  to  confirm  that  specification  requirements  are  satisfied  in  all  configurations 
and  conditions  that  will  be  encountered  during  the  mission.  Test  requirements  will  be  derived  from 
mission  analysis  and  analysis  of  potential  mission-critical  failures.  These  tests  can  be  integrated  with 
other  tests  but  may  have  to  be  performed  as  unique  tests  to  provide  required  test  conditions,  stresses, 
and  timelines  with  the  goal  of  exposing  mission  operations  flaws  as  early  as  possible. 

This  strategy  of  requirements  verification  through  multilevel  tests  depends  on  accurate  identification 
and  tracking  of  test  configurations  for  each  test  and  retest  through  all  the  test  levels.  Test 
configuration  identification,  traceability  of  lower  level  tests,  and  traceability  of  requirements  to  tests 
shall  be  documented  and  maintained  by  the  developer. 
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♦ 

♦ 


Developmental  Tests 


* 


Development  Tests 
Section  5.1,  5.4 


Design  verification/prototype 
COTS/reuse  evaluation 
HSI  validation 
Unit  test 

Unit/subsystem  integration  test 
Mission  operations  tests 
Performance  evaluation 
Stress/longevity  assessment 
Integrated  system  test 
Site  installation/checkout 


♦ 


♦ 


* 


Qualification/Acceptance 

Tests 

Section  5.2,  5.3 

Subsystem  Qualification 
System  Qualification 
Acceptance 
Site  Acceptance 


Transition  to  Operations 


Operational  Tests 
Section  5.5 


Prelaunch/Ops  Evaluation 
Operations  Readiness 
Postlaunch  Evaluation 


Sustainment 

Tests 


Same  as 

Developmental  Tests 


Retest,  Regression 
Tests 

Section  5.6 


* 


‘Increment,  Spiral,  Build  ... 


Figure  1.  Ground  system  test  phases. 
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4.2.1  Developmental  Tests 


Developmental  test  and  evaluation  (DT&E)  shall  be  conducted  to  assess  the  adequacy  of  ground 
system  designs,  verify  correct  implementation  of  designs  in  hardware  and  software  items,  and  verify 
compliance  with  requirements  as  the  system  will  be  used  in  operation  during  the  mission. 

Software,  developed  or  modified,  shall  be  tested  in  accordance  with  the  software  test  requirements  of 
this  standard.  Unmodified  commercial  vendor  software  embedded  in  hardware  components  shall  be 
tested  with  the  host  hardware  in  accordance  with  the  hardware  test  requirements  of  this  standard. 

1.  Development  Tests.  Tests  during  design  and  development  shall  be  conducted  to  identify 
problems  early  in  the  design  evolution,  assembly,  and  development  so  that  any  required 
corrective  actions  can  be  completed  prior  to  starting  formal  qualification  testing.  Tests  shall 
be  conducted  on  software  or  representative  hardware  articles  to  characterize  engineering 
parameters,  gather  data,  choose  a  design  approach,  verify  correct  implementation  of  designs, 
verify  interface  implementation,  and  verify  correct  operation  of  integrated  components  and 
subsystems  in  all  configurations  and  conditions  of  mission  operations.  Detailed  requirements 
for  development  tests  are  given  in  Section  5.1. 

2.  Qualification  Tests.  Qualification  tests  shall  be  conducted  to  verify  that  an  item  complies 
with  the  allocated  requirements  for  the  item.  Qualification  tests  shall  be  conducted  for  each 
specification  level  and  shall  include  COTS,  ND1  and  government  furnished  property  (GFP) 
items  that  are  allocated  program  requirements.  Items  that  are  similar  or  identical  may  be 
qualified  by  analysis.  Ground  hardware  items  may  be  deemed  qualified  by  similarity  if 
analyses  show  that  a  similar  unit  has  been  designed  and  tested  to  the  same  or  more  stressing 
environmental  and  life  performance  requirements  as  specified.  In  this  case,  relevant 
documentation  shall  prove  that  the  similarity  and  testing  equivalency  requirements  of  the 
procuring  activity  have  been  satisfied.  Detailed  requirements  for  qualification  tests  are  given 
in  Section  5.2. 

3.  Acceptance  Tests.  Acceptance  tests  shall  be  conducted  to  demonstrate  that  each  contract 
deliverable  item  meets  specification  requirements  and  to  identify  any  workmanship  flaws  or 
errors  in  manufacturing  and  assembly.  Selected  qualification  tests  performed  by  the 
developer  may  be  designated  as  acceptance  tests  by  the  procuring  activity,  but  acceptance 
tests  may  also  be  unique  tests  developed  and  performed  by  a  separate  agency  or  tester,  e.g., 
independent  test  organization,  user,  security  accreditation  agency.  Successful  completion  of 
the  acceptance  tests  allows  the  authorized  representative  of  the  procuring  activity  to  accept 
the  ground  system  in  partial  or  complete  performance  of  a  contract.  Detailed  requirements  for 
acceptance  tests  are  given  in  Section  5.3. 

4.  Integrated  System  Tests.  Integrated  system  tests  shall  be  performed  to  exercise,  the  total 
system,  integrated  with  interfacing  systems.  Unless  an  exception  is  approved  by  the  procuring 
activity,  integrated  system  tests  shall  be  performed  on  the  integrated  hardware  and  software 
items  installed  in  an  operational  system  with  connection  to  actual  interfaces,  conducted  at  the 
target  site  with  the  support  of  the  operational  personnel.  The  objective  of  the  tests  is  to  ensure 
that  the  products,  which  may  be  from  multiple  developers,  function  correctly  when 
integrated,  that  interfaces  are  verified,  and  that  all  system-level  requirements  or  specifications 
are  met.  In  addition,  end-to-end  tests  that  exercise  the  system  in  all  configurations  and 
conditions  of  mission  operations  shall  be  performed.  Detailed  requirements  for  integrated 
system  tests  are  given  in  Section  5.4. 
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4.2.2  Operational  Tests 


Operational  test  and  evaluation  (OT&E)  is  performed  by  the  procuring  activity  or  a  designated 
agency  in  coordination  with  the  operating  organization  to  appraise  a  system’s  operational 
effectiveness  and  suitability  of  new  items,  modifications,  or  installations  (hardware,  software,  or 
both)  and  provides  information  on  tactics,  doctrine,  organization,  and  personnel  requirements  in  the 
operational  environment.  Operational  tests  may  be  combined  with  developmental  tests  when 
objectives  can  be  satisfied  concurrently,  as  designated  by  the  procuring  activity.  Support  to 
operational  testing  is  provided  by  developers  as  designated  in  the  contract  statement  of  work  (SOW). 
Developers  collaborate  with  the  procuring  activity  test  team  and  independent  test  groups  to  conduct 
test  planning,  coordinate  and  align  DT&E  and  OT&E  objectives,  and  identify  potential  for  combined 
developmental  and  operational  tests.  Detailed  requirements  for  operational  tests  are  given  in 
Section  5.5. 

4.2.3  Retest  and  Regression  Test 

Retesting  shall  be  performed  if  a  test  discrepancy  or  test  item  failure  occurs  while  performing  any  of 
the  required  testing.  Following  correction  of  the  problem  identified  by  the  failure  analysis,  a  retest  of 
the  failed  test  case  and  related  areas  identified  by  analysis  of  the  implemented  fix  shall  be  performed. 
When  analysis  shows  previous  tests  have  been  invalidated  by  the  failure,  those  tests  shall  be  repeated. 
Regression  testing  of  software  shall  be  performed  after  any  modification  to  previously  tested  software 
and  after  each  installation  of  software  in  a  new  environment  or  at  a  new  site.  Detailed  requirements 
for  retest  and  regression  testing  are  given  in  Section  5.6. 

4.3  Test  Planning  and  Preparation 

The  test  plans,  descriptions,  and  procedures  provide  the  framework  for  identifying  and  interrelating 
all  of  the  individual  tests  and  test  procedures  needed. 

Test  planning  and  preparation  starts  with  requirements  analysis,  then  continues  in  each  program  phase 
to  ensure  that  testing  is  considered  in  developing  requirements  and  designs  and  that  test  plans  and 
procedures  completely  test  the  designs.  The  goal  is  to  create: 

•  Requirements  that  are  unambiguous,  complete,  and  testable 

•  Product  designs  that  include  provision  for  efficient  test,  e.g.,  start/restart  modes,  test  data 
injection,  built-in  test  equipment  (BITE),  and  test  software 

•  Product  designs  that  include  provisions  for  capture  of  test  data,  system  states,  and  test  results 

Planning  for  test  data  and  simulations  shall  ensure  that: 

•  Test  data  and  simulators  fully  cover  the  extent  of  operational  extremes  in  sufficient  quantity 
and  rates  to  test  worst-case  system  loads  and  responses 

•  Simulators,  simulations,  models,  and  test  tools  to  be  used  have  been  validated  to  ensure  that 
the  results  produced  and  degree  of  fidelity  are  adequate  for  the  intended  use  in  the  test 
program 

The  developer  shall  collaborate  with  integrated  test  teams  established  with  other  stakeholders  to 
ensure  that  the  combined  test  plans  and  procedures  provide  results  and  data  that  will  meet  the  test  and 
evaluation  needs  of  the  developer,  procuring  activity,  users,  and  operators. 

Note:  Ground  systems  for  space  programs  require  multilevel  test  plans  describing  the  hierarchical  test 
program,  generally  consisting  of  top-level  plans  for  the  overall  ground  system  and  subordinate  plans 
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for  each  element  of  the  system,  unit  test,  qualification  tests,  integration  tests,  acceptance  plan,  etc. 

The  test  plans  describe  implementation  of  the  test  phases  and  levels  illustrated  in  Figures  1  and  2.  In 
addition,  ground  software  is  usually  developed  using  incremental  approaches  with  multiple  software 
builds  tested  separately,  requiring  test  plans  that  describe  the  incremental  tests  and  the 
interrelationships  that  satisfy  the  overall  test  objectives  for  the  ground  system.  The  test  plans  on  large 
programs  may  be  phased  over  time,  with  details  completed  to  be  available  at  multiple  acquisition 
milestones. 

4.3.1  Test  Plans 

The  test  plans  shall  provide  a  general  description  of  each  test  planned  and  the  conditions  of  the  tests. 
The  test  plans  shall  be  based  upon  a  function-by-function  mission  analysis  and  any  specified  testing 
requirements.  Tests  shall  be  planned  and  executed  to  fulfill  test  objectives  from  development  through 
operations  to  the  maximum  extent  possible.  Test  objectives  shall  be  planned  to  verify  compliance 
with  the  design  and  specified  requirements  of  the  items  involved,  including  interfaces.  Software  test 
plans  shall  comply  with  the  Software  Development  Standard  for  Space  Systems  [10]. 

Test  plans  shall  describe  how  the  aggregate  tests  confirm  that  the  ground  system  performs  all 
requirements  using  realistic  operational  configurations,  conditions,  and  timelines.  The  plans  shall  also 
describe  and  provide  justification  for  any  exceptions  to  this  test  requirement  with  a  description  of  risk 
mitigation  for  the  exceptions. 

Test  plans  shall  include  or  refer  to  an  acceptance  plan  that  defines  the  acceptance  criteria,  the 
acceptance  processes,  and  the  documentation  required  to  obtain  agreement  by  the  procuring  activity 
that  the  acceptance  criteria  have  been  met. 

The  developer  shall  peer  review  test  plans  to  ensure  that  the  plans  are  complete,  compliant  with  this 
standard,  and  clearly  describe  the  test  to  be  performed. 

As  a  minimum,  a  test  plan  shall  address  the  following: 

1.  Identification  of  test  items,  e.g.,  names,  abbreviations  and  acronyms,  version  numbers, 
release  numbers. 

2.  System  overview  (or  reference)  describing  the  system  purpose,  functions,  operating  sites,  etc. 

3.  Overall  test  objectives  and  philosophy,  testing  strategy,  and  test  objective/expected  result  for 
each  item,  including  tailoring  or  interpretation  of  testing  requirements. 

4.  Identification  of  separate  test  sites  and  environments  (e.g.,  antenna  pads,  computer  rooms) 
and  the  specification  of  environments  and  site  requirements  or  constraints  for  each. 

5.  Required  special  test  tools  and  equipment,  simulators,  facilities,  interfaces,  and  any  needed 
downtime  of  facilities  and  equipment. 

6.  Identification  of  required  test  personnel,  organizations  and  the  roles  and  responsibilities  of 
each. 

7.  Planned  tests  and  test  levels  and  any  dependencies  on  other  tests. 

8.  Planned  exceptions  to  testing  in  actual  operational  configurations  and  conditions  and  risk 
mitigation  actions  for  those  exceptions. 

9.  Collection  or  development  of  required  test  data,  simulators,  and  test  drivers.  Identify  the  type 
and  quantity  of  test  data  sets  and  justification  of  adequacy  for  the  test  purpose.  Include  the 
plan  for  validating  simulators  and  test  drivers  prior  to  use  in  test. 
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10.  Sequence  of  tests  and  schedules  (or  reference)  for  tests  and  all  required  test  resources,  e.g., 
procedures,  data,  simulators,  tools,  facilities,  personnel,  showing  availability  for  preparation, 
start  and  duration  of  the  tests. 

1 1 .  Allocation  of  specification  requirements  to  appropriate  test  and  integration  levels.  Usually 
this  is  a  reference  to  a  requirements  traceability  matrix  listing  all  design  requirements  and 
indicating  a  cross-reference  to  a  verification  method  and  the  applicable  test  level. 

12.  Cecurity  requirements  for  the  test  facility  or  location,  if  any. 

4.3.2  Verification  Methods 

Assessment  of  each  requirement  and  the  design  to  meet  the  requirement  shall  be  performed  to 
determine  the  most  effective  verification  method(s).  The  verification  method(s)  selected  for  each 
requirement  shall  be  documented  in  a  verification  cross-reference  matrix  (VCRM),  to  be  used  in  test 
procedure  development.  The  four  requirement  verification  methods,  inspection,  analysis, 
demonstration,  and  test,  are  defined  in  Section  3. 

Tests  shall  be  conducted  using  documented  test  procedures  that  include  all  of  the  required  tests  to 
fulfill  with  the  test  objectives  in  the  approved  test  plans.  The  test  objectives,  testing  criteria,  and 
pass/fail  criteria  shall  be  stated  clearly  in  the  test  procedures.  The  test  procedures  shall  cover  all 
operations  in  enough  detail  so  that  there  is  no  doubt  as  to  the  execution  of  any  step.  Test  objectives 
and  criteria  shall  be  stated  to  relate  to  design  or  operations  specifications.  Traceability  shall  be 
provided  from  the  specifications  or  requirements  to  the  test  procedures  in  a  VCRM. 

The  developer  shall  peer  review  test  procedures  to  ensure  that  the  procedures  are  complete,  compliant 
with  this  standard,  and  clearly  define  the  test  steps  to  be  performed. 

Test  procedures  shall  include  a  measurement  tolerance  for  each  measured  parameter.  The  tolerance 
shall  include  a  positive  and  negative  uncertainty.  Where  practical,  the  individual  procedure  step  that 
satisfies  the  requirement  shall  be  identified.  The  test  procedure  for  each  item  shall  include,  as  a 
minimum,  descriptions  of  the  following: 

1.  Purpose,  objectives,  entry  criteria,  assumptions,  and  constraints,  including  which 
requirements  are  tested  by  specification  identifier. 

2.  Test  setup,  identifying  test  articles  by  configuration  identification,  including  software 
revision  for  software  under  test  and  supporting  software  such  as  operating  system. 

3 .  Initialization  requirements . 

4.  Input  data,  including  software  configuration  files,  etc. 

5.  Test  tools  and  test  instrumentation. 

6.  Expected  intermediate  test  results. 

7.  Requirements  for  recording  output  data. 

8.  Expected  output  data. 

9.  Minimum/maximum  requirements  for  valid  data  to  consider  the  test  successful  (at  procedure 
step  level,  where  appropriate). 

10.  Pass/fail  criteria  for  evaluating  results,  including  uncertainty  constraints  (at  procedure  step 
level,  where  appropriate). 

11.  Safety  considerations  and  hazardous  conditions. 
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4.4  Test  Environments  and  Facilities 


Test  environments  and  facilities  shall  be  equipped  and  configured  to  perform  the  tests  described  in  the 
approved  test  plans  and  procedures.  Test  environments  for  each  level  of  test  shall  be  planned  and 
developed  to  ensure  capacity  to  support  the  expected  usage  during  the  test  program.  Capacity  margin 
shall  be  adequate  to  maintain  the  test  schedule,  including  dry  runs,  test  run  for  record,  retests,  and 
regression  tests.  Test  environment  spares  shall  be  sufficient  for  testing  to  proceed  while  failed 
equipment  is  replaced  or  repaired. 

4.5  Test  instrument  Accuracy  and  Calibration 

Equipment  and  instruments  used  in  tests  shall  be  sufficiently  accurate  to  measure  within  ±‘/2  of  the 
error  interval  specified  for  the  measurand.  Each  instrument  shall  be  used  within  the  tolerances  and  the 
operating  environment  specified  in  the  instrument  specifications  provided  by  the  manufacturer. 
Instruments  that  require  periodic  recalibration  shall  have  been  recalibrated  in  accordance  with  the 
developer’s  metrology  plan,  but  at  no  greater  interval  than  recommended  by  the  instrument 
manufacturer. 

4.6  Documentation 

Test  plans,  procedures,  and  the  test  documentation  described  below  shall  be  collected  and  maintained 
in  test  documentation  files.  Developers  shall  maintain  the  test  documentation  files  for  the  duration  of 
development,  operations  and  maintenance. 

4.6.1  Test  Configuration  Audit 

For  each  test  execution,  a  test  configuration  audit  shall  be  performed  to  document  the  versions  and 
revision  designators  of  all  hardware,  software,  operating  systems,  configuration  files  and  other  items 
constituting  the  test  environment.  Test  equipment  used  shall  be  listed  with  calibration  dates  and 
accuracy. 

4.6.2  Test  Data 

Test  data  shall  include  the  data  required  for  preparation,  simulation,  or  configuration,  for  performing 
the  test,  and  data  collected  during  and  after  the  test.  Test  data  shall  be  collected  and  maintained  in  a 
form  to  permit  retest  and  the  evaluation  of  performance  under  the  various  specified  test  conditions. 

4.6.3  Test  Log 

Formal  tests  shall  be  documented  in  a  test  log.  The  test  log  shall  identify  the  personnel  involved  and 
be  time -tagged  to  permit  a  reconstruction  of  test  events  such  as  start  time,  stop  time,  anomalies, 
procedure  steps  changed  or  not  completed,  and  any  periods  of  interruption. 

4.6.4  Test  Discrepancies 

Anomalies,  discrepancies,  and  failures  occurring  during  test  activities  shall  be  documented  and 
dispositioned  as  specified  in  the  developer’s  quality  control  plan.  Test  discrepancy  and  resolution 
records  shall  be  reported  to  the  procuring  activity  as  required  in  applicable  contract  or  development 
agreement.  Discrepancy  metrics  must  be  collected  and  analyzed  to  assess  testing  effectiveness  as 
described  in  Section  5.7. 


18 


4.6.5  Qualification  and  Acceptance  Test  Report 

For  qualification  and  acceptance  tests,  test  results  shall  be  documented  in  test  reports.  The  test  report 
shall  state  the  degree  of  success  in  meeting  the  test  objectives  and  shall  document  the  details  and 
conclusions  from  the  test  results  (including  verification  status  of  each  requirement),  and  a  summary  of 
the  test  results,  deficiencies,  problems  encountered,  and  problem  resolutions.  The  responsible 
developer  design  engineer  shall  certify  the  accuracy  of  the  results. 
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5.  Detailed  Requirements 


Detailed  requirements  are  provided  for  developmental  tests  and  evaluation  (DT&E)  and  operational 
tests  and  evaluation  (OT&E).  Developmental  tests  include  tests  performed  during  development  of  a 
ground  system  to  support  design  and  development,  verify  specification  requirements  and  interface 
implementation,  demonstrate  acceptability  of  contract  end  items,  and  certify  readiness  for  operational 
test  by  the  procuring  activity  to  validate  the  required  capabilities  of  the  system. 

5.1  Development  Tests 

Development  tests,  including  hardware  parts,  subassembly  and  unit,  software  unit,  software 
component  integration,  and  hardware/software  integration  testing  shall  be  conducted  to: 

1 .  Validate  new  design  concepts  and  alternative  designs  or  the  application  of  proven  concepts 
and  techniques  to  a  new  configuration. 

2.  Assist  in  the  evolution  of  designs  from  the  conceptual  phase  to  the  operational  phase. 

3.  Verify  that  implemented  software  and  assembled  hardware  is  in  accordance  with  the  design. 

4.  Verify  specification  requirements  are  satisfied  in  the  actual  conditions  of  mission  operations, 
exceptions  documented  and  approved  by  the  procuring  activity. 

5.  Reduce  the  risk  of  committing  designs  to  fabrication  and  production. 

6.  Validate  design  changes. 

7.  Develop  and  validate  qualification  and  acceptance  test  procedures. 

8.  Investigate  problems  or  concerns  that  arise  after  successful  qualification. 

Note:  The  amount  and  type  of  development  testing  depend  upon  the  maturity  of  the  subsystems  and 
units  used  and  the  operational  requirements  of  the  specific  program.  An  objective  of  development 
testing  is  to  identify  problems  early  in  the  design  evolution  so  that  any  required  corrective  actions  can 
be  completed  prior  to  starting  formal  qualification  testing. 

In  addition  to  testing  the  functional,  performance,  accuracy  and  timeliness  objectives  of  a  design, 
development  tests  shall  be  performed  to  confirm  manufacturability,  compatibility  with  system  safety, 
and  nonfunctional  quality  attributes  that  are  stated  in  the  applicable  specifications  such  as: 

•  Interoperability 

•  Reliability/Availability/Maintainability 

•  Scalability 

•  Testability 

•  Usability 

•  Supportability 

System  and  information  security  requirements  are  included  in  functional  requirements  and  tests  shall 
be  conducted  to  validate  the  design  and  implementation,  plus  any  specified  extra  security  test 
requirements,  e.g.,  tests  by  security  accreditation  agencies. 

Development  tests  to  identify  performance  break  points  or  margins  beyond  specification  limits  shall 
be  performed  if  such  tasks  are  defined  in  the  SOW  or  special  test  requirements. 
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Ground  hardware  development  tests  may  be  conducted  on  breadboard  equipment,  prototype 
hardware,  or  representative  operational  equipment.  Software  development  tests  may  be  conducted  in 
a  software  development  environment  or  a  software  integration  and  test  environment. 

Development  test  procedures,  input  data,  results,  logs,  required  test  metrics,  deficiencies,  and 
corrective  actions  shall  be  recorded  in  the  Software  Development  File  for  software  unit  level  tests,  in 
Test  Documentation  Files  for  hardware  and  other  software  tests.  Errors  and  defects  discovered  in 
development  tests  shall  be  documented  and  corrected  in  accordance  with  the  developer’s  quality 
assurance  procedures. 

Development  tests  may  be  conducted  at  developer’s  in-plant  test  facilities,  at  a  government -approved 
testbed,  or  at  any  other  appropriate  test  facility.  However,  when  performed  at  a  government  facility, 
that  facility  may  require  approval  of  the  test  plans  and  procedures.  Internal  developer  documentation 
of  development  test  plans,  test  procedures,  and  test  results  are  normally  used  unless  stated  otherwise. 

5.1.1  Hardware 

Hardware  development  tests  shall  be  performed  to  achieve  the  objectives  listed  above  for  antennas, 

RF  subsystems,  electronics,  mechanical  mechanisms,  and  related  components. 

Tests  should  be  planned  as  progressive  steps  to  identify  problems  early  in  design  and  assembly  so 
corrective  actions  can  be  completed  before  starting  formal  qualification  testing.  Hardware 
development  tests  shall  be  performed  at  part,  unit,  the  hardware  integration  levels  (subassembly, 
subsystem,  etc.),  and  hardware/software  integration  levels.  In  addition  to  development  tests  of 
functional  and  performance  requirements,  compliance  with  applicable  site  or  facility  installation 
requirements  shall  be  verified. 

5. 1.1.1  Hardware  Test  Levels 

Part,  Material  and  Process  Tests.  These  tests  shall  be  conducted  to  qualify  parts,  materials,  and 
processes  to  ensure  proper  application  in  the  design,  to  ensure  adequate  performance  margins,  and  to 
develop  acceptance  criteria  for  parts  and  material  to  avoid  assembling  defective  hardware 
components.  As  a  minimum,  all  new  types  of  parts,  materials,  and  processes  are  tested. 

Subassembly  and  Unit  Tests.  Subassemblies  and  units  shall  be  tested  to  demonstrate  feasibility, 
minimize  design  risk,  and  evaluate  design  and  manufacturing  alternatives  and  tradeoffs  required  to 
best  achieve  the  development  objectives.  Tests  are  also  conducted  to  develop  in-process 
manufacturing  or  assembly  tests,  inspections,  and  acceptance  criteria  for  subassemblies  and  units  to 
avoid  assembling  defective  hardware  components. 

Subsystem  Tests.  Subsystem  tests  shall  be  conducted  to  evaluate  components,  demonstrate 
compliance  of  subsystem  requirements  flowed  down  to  subsystem  specifications  from  the  system- 
level  requirements,  and  provide  baseline  data  on  subsystem  performance  that  allow  trending  and 
performance  comparisons  when  system  components  are  replaced  and/or  upgraded. 

5. 1.1. 2  Environmental  and  Specialty  Tests 

Environmental  and  specialty  tests  shall  be  performed  to  verify  the  capability  of  hardware  to  operate 
reliably  in  the  installed  environment  and  perform  required  functions  with  safety  for  operating 
personnel. 

Environmental  Tests.  These  tests  shall  be  conducted  to  verify  that  specified  functional,  performance, 
storage,  and  relocation  requirements  are  satisfied.  Hardware  operated,  stored,  or  relocated  in 
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environmentally  controlled  environments  can  generally  be  verified  by  analysis  and/or  similarity. 
However,  tests  of  hardware  required  to  be  used  in  one  or  more  of  the  environmental  conditions  listed 
in  this  section  shall  be  performed  to  verify  compliance  with  functional  and  performance  requirements 
Similarly,  if  storage  or  transport  is  required  in  one  or  more  of  the  environmental  conditions  listed, 
testing  is  required  to  demonstrate  suitability  for  return  to  operations  after  being  stored  or  relocated  in 
those  conditions.  The  environmental  conditions  are  as  follows: 


•  Hail 

•  Sand/blowing  sand 

•  Dust 

•  Mud 

•  Salt  fog 

•  Solar  radiation 

•  Humidity 

•  Fungus  growth 

•  Altitude 

•  Wind  speed 


Snow  and/or  ice  loading 

Temperature 

Chemical  attack 

Thermal  shock 

Handling  shock/drop  distance 

Electromagnetic  pulse 

Immersion 

Vibration 

Explosive  atmosphere 
Biological  attack  and/or  infestation 


Environmental  testing  shall  be  performed  in  accordance  with  M1L-STD-810  [2],  Testing  for  high- 
altitude  electromagnetic  pulse  protection  shall  be  performed  in  accordance  with  M1L-STD-188-125-1 
[3]  for  fixed  items  and  M1L-STD-188-125-2  [4]for  non-fixed  items. 


Electromagnetic  Compatibility  (EMC)  Tests.  Ground  hardware  shall  be  verified  to  be  compliant 
with  FCC  Class  A  requirements  in  Federal  Code  of  Regulations  FCC  Part  15  [8]  as  a  minimum. 
Verification  may  be  by  certification  from  equipment  vendors.  If  a  vendor  does  not  certify  compliance 
with  FCC  Class  A  requirements,  or  subsystem  specifications  have  EMC  requirements  exceeding  FCC 
Class  A  requirements,  tests  and  measurements  for  verification  shall  be  conducted  in  accordance  with 
standards  designated  by  the  procuring  activity  or  M1L-STD-461  [1]. 

Power  Tests.  For  each  subsystem,  rack,  or  device  that  connects  to  the  facility  power  infrastructure  or 
is  powered  from  an  interfacing  subsystem,  power  consumption  and  total  harmonic  distortion  shall  be 
measured  at  both  peak  load  and  nominal  operating  load.  Measurements  shall  be  performed  in 
accordance  with  standards  designated  by  the  procuring  activity  or  IEEE  Standard  1100  [8]. 

Grounding  and  Bonding  Tests.  Grounding  systems  shall  be  tested  for  electrical  resistance  and 
continuity.  Measurements  shall  be  performed  in  accordance  with  standards  designated  by  the 
procuring  activity  or  IEEE  Standard  1100  [8]. 

Fault  Management  and  Built-in  Test  Equipment  (BITE)  Tests.  Fault  Management  and  BITE 
capabilities  shall  be  demonstrated  to  verify  identification  of  failures  to  a  lowest-replaceable-unit  level. 
BITE  capability  to  resolve  specified  multiple  system  failures  shall  be  demonstrated.  Operation  of 
BITE  shall  be  demonstrated  for  both  primary  and  redundant  components.  Failover  to  redundant 
components  to  recover  from  detected  faults  shall  be  demonstrated,  and  required  failover  time  verified. 
Manual  initiation  of  fault  detection  and  isolation  shall  be  demonstrated.  The  ability  to  select 
redundant  components  by  command  and  restore  system  operation  shall  be  demonstrated.  BITE  shall 
be  verified  at  subsystem  level,  when  possible,  or  at  integrated  system  level  if  necessary  to  provide 
required  configurations  for  test. 

Dependability  Tests.  Verification  of  reliability,  availability,  and  maintainability  and  integrity 
requirements  shall  be  demonstrated.  Demonstrations  may  be  conducted  over  time  periods  shorter 
than  those  required  and  extrapolated  to  cover  the  required  time  period.  Dependability  testing  shall  be 
conducted  using  operational  scenarios  provided  or  approved  by  the  procuring  activity.  Protection 
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from  hazards  designated  by  the  procuring  activity  to  personnel  and  to  the  system  and  the  facility  into 
which  the  hardware  is  to  be  incoiporated  shall  be  verified.  Testing  of  RF  exposure  shall  be  performed 
in  accordance  with  standards  designated  by  the  procuring  activity  or  IEEE  Standard  C95[7]. 

Non-fixed  Hardware — Unique  Tests.  Hardware  for  non-fixed  systems  will  have  additional 
requirements  not  associated  with  fixed-site  systems.  Unique  tests  and  demonstrations  required  to 
verify  these  requirements  may  include,  but  not  necessarily  be  limited  to,  the  following: 

1 .  Setup  and  teardown  time  with  a  specified  number  of  people 

2.  Setup  and  teardown  time  in  specific  climatic  environments 

3.  Transportability 

a.  Compatibility  with  specific  aircraft,  ships,  other  vehicles 

b.  Compatibility  with  railway  transport  in  specific  countries 

c.  Compatibility  with  specific  road  conditions  and  speeds 

4.  Compatibility  with  cargo  handling  equipment  and/or  facilities 

5.  Permitted  range  of  motion  during  operations 

6.  Operations  in  other  than  a  horizontal  position 

Specialized  fixtures,  facilities  and  access  to  external  equipment  shall  be  identified  in  advanced  test 
planning  and  coordinated  with  the  procuring  activity. 

5. 1.1. 3  Commercial  or  Non-development  Item  (NDI)  Hardware  Test. 

Testing  of  commercial  or  NDI  hardware  shall  be  conducted  to  verify  that  the  function  and 
performance  of  the  item  is  as  specified  in  vendor  documentation  and  meets  requirements  for  use  in 
the  current  system  design.  These  tests  may  be  performed  as  part  of  a  selection  process  or  to  verify 
functional  capabilities  and  performance  of  an  existing  commercial  or  NDI  item.  Developer-selected 
acceptance  tests  shall  be  conducted  on  delivered  items  to  ensure  that  they  conform  to  the  same 
specifications  as  those  tested  during  the  selection  process.  Tests  of  general-purpose  computer 
hardware  may  be  performed  as  unique  hardware  function  and  performance  tests  or  in  conjunction 
with  tests  of  software  executed  on  the  computer.  Required  capabilities,  characteristics  or  features  not 
described  in  vendor-furnished  documentation  shall  be  tested  as  if  the  hardware  is  a  developmental 
item. 

5.1.2  Software 

Software  development  tests  shall  be  performed  to  achieve  the  objectives  listed  in  Section  5.1  for 
software  and  firmware.  Tests  shall  be  planned  as  progressive  steps  to  identify  problems  early  in 
design  and  implementation  so  corrective  actions  can  be  completed  before  formal  qualification  testing 
is  started. 

Software  development  tests  shall  be  performed  at  unit,  unit  integration,  and  hardware/software 
integration  levels.  Development  tests  for  software  include  prototype  tests,  unit  tests,  COTS  or  reuse 
selection/verification  tests,  performance  tests,  integration  tests,  and  interface  tests.  Software  tests  shall 
comply  with  test  requirements  in  Software  Development  Standard  for  Space  Systems  [6], 

5.1.3  Integration  Test 

Tests  shall  be  performed  to  verify  the  integration  of  subsystems  and  operational  hardware,  correct 
implementation  of  interfaces,  and  error-free  interaction  between  subsystems  and  between  software 
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and  hardware.  These  tests  are  performed  in  the  development  facilities,  starting  with  earliest 
integration  steps,  whereas  integrated  system  tests  (see  Section  5.4)  are  performed  at  operational  sites 
or  a  designated  operational  testbed,  after  integration  of  the  system  under  development  is  completed. 

Unless  exceptions  are  approved  by  the  procuring  activity,  tests  shall  be  performed  with  software  and 
hardware  of  the  system  in  operational  configuration(s),  including  all  reuse  software  and  hardware, 
modified  and  unmodified,  and  COTS  software  and  hardware.  Approved  exceptions  are  to  be 
documented  in  the  test  report.  As  a  minimum,  integration  test  cases  shall  include  testing: 

1.  Software-to-software,  software-to-hardware,  hardware-to-hardware  interfaces,  including 
limits  and  boundaries  in  interface  control  documents 

2.  Software  and  hardware  functional  and  performance  requirements  allocated  to  the  integrated 
system  level 

3.  Integrated  error  and  exception  handling 

4.  System-level  performance,  such  as  acquisition  and  tracking,  system  throughput,  timing,  and 
accuracy  requirements 

5.  Stress  testing  of  worst-case  scenario(s),  data  and  transaction  loads 

6.  Startup,  termination,  and  restart  (when  applicable) 

7.  Fault  detection,  isolation  and  recovery  handling  (e.g.,  fault  tolerance,  failover  to  redundant 
units,  BITE  operation,  failure  data  capture  and  reporting) 

8.  Resource  utilization  measurement  (e.g.,  CPU,  memory,  storage,  bandwidth) 

9.  Longevity  and  endurance  tests  (day/week-in-the-life)  of  sufficient  length  to  demonstrate 
required  stability  and  reliability 

Note:  Large  ground  systems  consist  of  multiple  architectural  levels  e.g.,  elements,  or  segments.  Such 
systems  may  require  multiple  levels  of  integration  tests.  Strategy  and  planning  for  tests  at  multiple 
levels  are  to  be  described  in  the  ground  system  integration  plans. 

5.2  Qualification  Test 

Qualification  tests  shall  be  performed  to  verify  requirements  allocated  in  ground  system 
specifications.  Tests  shall  be  performed  for  requirements  in  hardware  and  software  specifications  in 
the  ground  system  specification  tree.  Test  procedures  shall  be  developed  for  each  test  case  indentified 
in  the  system  verification  test  plan.  Qualification  tests  shall  be  performed  with  approved  test 
procedures  and  with  hardware  configurations  and  software  versions  identified,  controlled,  and 
installed  by  independent  configuration  management  staff. 

A  test  readiness  review  (TRR)  shall  be  conducted  prior  to  test  execution  to  verify  concurrence  of 
required  stakeholders  that  the  test  procedures,  environment,  equipment,  and  personnel  are  at  a  state  of 
readiness  so  that  the  test  objectives  will  be  satisfied  by  the  test.  Dry  run(s)  of  the  tests  shall  be 
conducted  to  confirm  that  the  procedures,  equipment,  and  test  environment  are  ready  for  successful 
formal  run-for-record  (RFR).  If  not,  corrective  actions  shall  be  performed  and  test  procedure  updates 
made  and  dry  runs  repeated.  Following  the  RFR,  a  test  review  board  (TRB)  shall  be  conducted  to 
confirm  that  the  test  objectives  were  met,  review  test  discrepancies,  and  identify  any  retest  required. 
Test  results  shall  be  documented  in  a  test  report(s)  and  shall  include  a  record  of  resolution  of  any  test 
discrepancies.  TRR,  and  TRB  shall  be  conducted  in  accordance  with  the  technical  review  and  audit 
standard  specified  by  the  procuring  activity. 
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Hardware.  Hardware  qualification  tests  shall  be  performed  to  verify  requirements  allocated  to 
hardware  unit  and  subsystem  specifications  as  well  as  to  ensure  that  subsequent  items  produced  and 
assembled  from  the  same  design  meet  the  same  requirements  as  the  qualification-tested  items. 

Hardware  with  multiple  items  from  the  same  design  may  be  qualification  tested  using 
protoqualification  methods,  i.e.,  a  first  article  is  tested  to  full  specification  levels  plus  a  margin  to 
ensure  compliance  given  expected  variation  in  manufacturing  and  assembly  of  subsequent  items. 
Subsequent  hardware  items  can  then  be  qualified  by  similarity. 

Assessment  of  first  article  manufacturing/assembly  shall  be  performed  to  identify  potential 
manufacturing  or  workmanship  issues  that  would  result  in  noncompliance  in  subsequent  items.  These 
issues  shall  be  assessed  in  acceptance  tests  and  inspections  for  each  item.  In  production,  random 
testing  and  inspection  shall  be  performed  to  provide  assurance  of  compliance. 

COTS  hardware  operating  within  the  vendor-specified  operating  range  shall  be  qualification  tested 
for  functional  and  performance  requirements.  Compliance  with  environmental  requirements  may  be 
verified  by  analysis. 

General  purpose  computers  may  be  qualification  tested  in  conjunction  with  software  qualification 
tests. 

Hardware  with  embedded  software/firmware  shall  be  qualification  tested  following  qualification  of 
the  software/firmware  unless  the  hardware/software  architecture  requires  test  as  an  integrated  entity. 

Software.  Software  qualification  tests  shall  be  performed  to  verify  requirements  allocated  to  software 
requirements  specifications  and  software  interface  specifications.  Software  qualification  tests  shall 
comply  with  test  requirements  in  Software  Development  Standard  for  Space  Systems  [6], 

Ground  System.  Qualification  tests  shall  be  performed  to  verify  ground  system  specification 
requirements.  The  tests  shall  be  performed  in  actual  operational  configurations  and  conditions  to 
verify  compliance  with  specifications  as  the  system  will  be  used  in  mission  operations.  Exceptions 
and  risk  mitigation  shall  be  approved  by  the  procuring  activity.  These  tests  may  be  designated  as 
acceptance  tests  in  lieu  of  separate  acceptance  tests,  as  determined  by  the  procuring  activity  (see 
Section  5.3).  Ground  system  requirements  that  are  completely  allocated  to  a  single  lower-level 
specification  may  be  considered  verified  in  the  qualification  test  for  that  lower-level  specification  and 
need  not  be  reverified  at  system  level.  Requirements  that  are  partitioned  and  allocated  to  multiple 
lower-level  specifications  shall  be  verified  in  system  qualification  tests.  Rationale  for  qualification  at 
lower  level  shall  be  described  in  appropriate  test  plans  and  indicated  in  the  VCRM. 

Large  ground  systems  that  have  multiple  architectural  levels,  e.g.,  elements  or  segments,  will  have 
multiple  levels  of  qualification  tests.  Strategy  and  planning  for  tests  at  multiple  levels,  and 
interrelationships  of  the  tests,  shall  be  described  in  the  ground  system  test  plans.  Allocation  of 
requirements  to  test  levels  shall  be  documented  in  the  VCRM. 

5.3  Acceptance  Test 

Acceptance  tests  shall  be  conducted  to  demonstrate  the  acceptability  of  each  deliverable  item  to  meet 
specification  requirements  and  to  demonstrate  error-free  workmanship  in  manufacturing  and 
assembly.  Acceptance  tests  may  be  unique  tests  or  a  subset  of  qualification  tests.  The  acceptance  tests 
and  the  procedures  for  the  acceptance  tests  shall  be  approved  by  the  procuring  activity.  Acceptance 
tests  may  be  performed  by  the  developer  or  an  independent  organization  as  designated  by  the 
procuring  activity.  The  scope,  organization,  roles,  responsibilities,  sequence,  and  location  of 
acceptance  tests  shall  be  as  described  in  the  acceptance  test  plan. 
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The  acceptance  test  hardware  and  software  baseline  shall  include  in  the  final  operational 
configuration  all  development  items,  ND1,  GFP,  and  COTS  hardware  and  software.  Factory 
acceptance  test  (FAT)  shall  be  performed  to  verify  all  system  requirements,  if  possible.  Requirements 
that  require  site  configuration  or  environments  to  verify  shall  be  tested  during  site  acceptance  tests  or 
unique  tests  to  verify  those  specific  requirements. 

Site  acceptance  tests  (SAT)  shall  be  performed  by  the  developer  in  accordance  with  site-approved 
procedures  to  verify  compliance  of  the  system  with  site  installation  requirements  and  ensure  that 
integration  of  the  delivered  system  into  the  site  configuration  will  cause  no  disruption  to  operations. 
Site  installation  tests  shall  be  performed  by  the  developer  prior  to  SAT  to  verify  integrity  of  hardware 
following  transportation  and  installation,  and  to  verify  software  compatibility  with  site  network  and 
other  interface  configurations.  Site  installation  tests  may  be  selected  factory  tests  or  unique  tests 
designed  to  adequately  confirm  readiness  for  site  acceptance  tests,  integrated  system  tests,  and 
operational  tests. 

The  TRR,  dry  run(s),  RFR,  and  TRB  shall  be  conducted  and  test  report(s)  prepared  as  described  in 
Section  5.2,  Qualification  Test.  A  designated  representative  of  the  procuring  activity  will  be  present  at 
TRR,  RFR,  and  TRB. 

Acceptance  test  results  shall  be  documented  as  required  in  the  acceptance  plan  for  use  in  the  physical 
configuration  audit  (PCA)  and  system  verification  review  (SVR)  or  functional  configuration  audit 
(FCA). 

5.4  Integrated  System  Tests 

Integrated  system  tests  shall  be  performed  to  exercise,  to  the  maximum  extent  that  is  practical  and 
possible,  the  system  in  development  plus  all  systems  that  interface  and  interoperate  with  that  system. 
Unless  exceptions  are  approved  by  the  procuring  activity,  integrated  system  tests  shall  be  performed 
on  the  integrated  hardware  and  software  items  installed  in  an  operational  system  with  connection  to 
actual  interfaces,  and  these  tests  shall  be  conducted  at  the  target  site  with  the  support  of  the 
operational  personnel. 

The  objective  of  the  tests  is  to  ensure  that  the  products,  even  if  from  multiple  developers,  function 
correctly  when  integrated,  that  interfaces  are  verified,  and  that  all  system  requirements  or 
specifications  are  met.  Tests  shall  be  designed  to  use  actual  mission  operations  configurations  and 
conditions  to  the  maximum  practical  extent.  End-to-end  tests  shall  be  performed  that  exercise  the  full 
operational  configuration,  including  space  vehicle,  with  operational  timelines  and  data  loads.  External 
systems  that  could  affect  the  operation  of  system(s)  in  test,  e.g.,  RF  emitters,  shall  be  operated  or 
simulated  to  replicate  conditions  expected  during  operations.  Test  articles,  configurations  and 
conditions  that  differ  from  the  operational  configuration  shall  be  identified  in  test  planning  and  risk 
mitigation  described. 

A  development  test  bed  approved  by  the  procuring  activity  as  sufficiently  simulating  the  operational 
system  capability  for  test  puiposes  may  be  used  for  integrated  system  tests  if  target  sites,  operational 
complexes,  or  other  suitable  operational  support  areas  are  not  available. 

The  integrated  system  tests  shall  incorporate  tests  of  the  affected  interfaces  of  the  ground  equipment 
and  software  with  other  elements  of  the  operational  system.  The  tests  shall  be  structured  as 
appropriate  to  demonstrate  design  requirements  of  the  system  related  to  such  items  as  performance, 
electromagnetic  compatibility,  reliability,  maintainability,  system  safety  (e.g.,  hazardous  noise, 
radiation  hazards,  pressure  vessels),  logistics  supportability,  operational  procedures,  and  personnel 
performance. 
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The  tests  shall  demonstrate  the  following,  as  applicable  to  the  installation,  modification,  or  upgrade: 

1 .  reliable  operation  is  achieved  at  specified  design  limits. 

2.  system  functional  and  performance  requirements  are  met. 

3.  the  system  can  recover  from  hardware  or  software  malfunctions  within  the  specified  time 
without  loss  of  data  or  control. 

4.  performance  requirements  are  met  under  all  required  logical  or  physical  device  assignment 
combinations. 

5.  software  and  hardware  modifications  or  upgrades  have  not  degraded  the  capability  of  the 
system’s  baseline  or  of  other  operational  systems. 

6.  security  mechanisms  are  in  place  or  incorporated  to  protect  resources  from  unauthorized 
access  or  break-in. 

Tests  are  to  be  focused  on  the  external  interfaces  involved,  the  use  of  operational  databases  and 
operational  scenarios,  and  the  system  requirements  from  a  mission  operations  perspective. 

The  tests  shall  include  other  applicable  tests,  such  as: 

1 .  A  reliability  demonstration 

2.  A  maintainability  demonstration 

3.  System  safety  tests,  inspections,  and  evaluations  in  such  areas  as  hardware  inspections  for 
electrical  and  mechanical  hazards,  including  caution  labeling 

4.  Evaluation  of  the  fire  suppression  system 

5.  Evaluation  of  emergency  systems 

6.  Use  of  hazardous  materials 

7.  Possibility  of  personnel  exposure  to  any  equipment  and  conditions  considered  hazardous 

8.  RF  radiation  testing  to  determine  actual  levels  of  radiation  to  which  personnel  may  be 
exposed  and  to  evaluate  the  accuracy  of  the  mathematical  predictions  of  radiation  levels 

9.  Proper  functioning  of  any  radiation  warning  systems 

10.  Proper  procedures  for  inspection,  operation,  and  maintenance  of  pressure  vessels 

The  TRR,  dry  run(s),  RER,  and  TRB  shall  be  conducted  and  test  report(s)  prepared  as  described  in 
Section  5.2,  Qualification  Test.  A  designated  representative  of  the  procuring  activity  shall  be  present 
at  TRR,  RER  and  TRB. 


5.5  Operational  Tests 

Operational  tests  are  planned  and  conducted  by  the  procuring  activity  test  organization  (or  an 
independent  test  organization  designated  by  the  procuring  activity),  in  coordination  with  the  operating 
organization.  The  test  organization  is  responsible  for  the  development  of  test  plans,  test  tools,  and  test 
procedures,  the  conduct  of  the  test,  and  preparation  of  test  reports.  Developers  shall  provide  support 
for  the  tests  and  the  resolution  of  deficiencies  within  the  scope  of  the  applicable  contract  or 
development  agreement  and  as  directed  by  the  procuring  activity.  Support  by  developers  includes 
collaboration  with  the  procuring  activity  test  team,  the  operating  organization  and  independent  test 
groups  to  conduct  test  planning,  coordinate  and  align  DT&E  and  OT&E  objectives,  and  identify 
potential  combined  developmental  and  operational  tests. 
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5.6  Retest  and  Regression  Tests 

5.6.1  Retest 

When  previously  tested  hardware  parts/components  or  software  is  changed,  the  hardware  and 
software  shall  be  retested.  Retesting  may  also  be  necessary  if  a  discrepancy  occurs  while  performing 
required  testing.  Limited  retesting  may  be  adequate  to  verify  that  the  change  is  satisfactory  and  no 
new  problems  have  been  introduced.  However,  the  extent  of  retest  shall  be  determined  by  evaluation 
of  the  failure  analysis,  cause  of  the  failure,  and  extent  of  the  changes. 

Retest  During  Qualification/Acceptance  Testing.  If  a  test  discrepancy  occurs  during  qualification 
or  acceptance  testing,  the  test  may  be  continued  without  corrective  action  if  the  discrepancy  does  not 
affect  the  validity  of  test  data  obtained  by  the  continuation  of  testing.  Otherwise,  the  test  shall  be 
interrupted,  the  cause  of  the  discrepancy  determined,  and  corrective  action  completed  before  testing  is 
resumed.  If  the  discrepancy  is  caused  by  the  test  setup,  test  software,  or  a  failure  in  the  test 
equipment,  the  test  being  conducted  at  the  time  of  the  failure  may  be  continued  after  the  cause  is 
removed  and  corrected  as  long  as  the  failure  did  not  overstress  the  item  under  test.  Retesting  may  be 
required  to  establish  a  basis  for  determining  compliance  of  a  test  item  to  a  specification  or 
requirement,  and  may  be  required  to  assess  the  readiness  of  test  items  for  subsequent  testing. 

Retest  During  Operational  Testing.  If  a  discrepancy  occurs  during  operational  testing,  the  test 
director  designated  by  the  procuring  activity  shall  be  responsible  for  assessing  the  effect  of  the 
discrepancy  to  determine  whether  the  discrepancy  has  jeopardized  the  probable  success  of  the 
remainder  of  the  test.  The  test  director  may  decide  to  continue,  or  halt  the  test  and  conduct  a 
subsequent  retest.  Developers  shall  provide  support  for  resolution  of  deficiencies  and  subsequent 
retest  within  the  scope  of  the  applicable  contract  or  development  agreement  and  as  directed  by  the 
procuring  activity. 

5.6.2  Software  Regression  Test 

To  identify  unexpected  side  effects  of  software  changes,  software  shall  be  regression  tested  in 
addition  to  tests  of  the  specific  changes.  Regression  tests  shall  be  performed  (1)  after  changes  to 
previously  tested  software,  and  (2)  after  installation  of  modified  software  in  new  environments,  e.g., 
new  test  environments,  site  reinstallation,  new  sites.  Software  regression  tests  shall  comply  with  test 
requirements  in  Software  Development  Standard  for  Space  Systems  [6], 

The  scope  of  regression  tests  shall  be  determined  by  analysis  of  the  type  and  extent  of  the  changes. 
The  rationale  for  selection  and  scope  of  regression  tests  shall  be  documented  in  test  documentation 
files.  Considerations  for  the  extent  of  regression  test  include  the  structure  of  the  software  (tightly  or 
loosely  coupled),  criticality  of  the  software  to  operations,  complexity  of  functionality,  complexity  of 
interfaces,  and  extent  of  the  modifications  if  changing  a  mature  system. 

Regression  tests  may  be  selected  subsets  of  previous  compliance  tests  or  unique  tests  designed  to 
probe  for  unexpected  side  effects  of  changes  or  installation  in  new  environments,  e.g.,  standard 
regression  tests  run  on  a  software  component  after  any  change,  or  installation  checkout  tests. 

5.7  Test  Discrepancy  Management  and  Reporting 

Testing  of  complex  ground  system  hardware  and  software  can  result  in  a  large  numbers  of 
discrepancy  reports,  requiring  disciplined  tracking  and  management  to  closure  and  accounting  of  the 
disposition  of  each  discrepancy.  Each  test  discrepancy  shall  be  recorded  in  a  discrepancy  report  (DR) 
and  corrective  actions  are  to  be  documented.  Anomalous  behavior  observed  in  non-test  operation  of 
the  system,  such  as  demonstrations,  rehearsals,  etc.,  shall  be  recorded  and  resolved  in  the  same 
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manner  as  test  discrepancies.  Each  discrepancy  report  shall  include,  minimally,  a  description  of  the 
problem,  its  severity  (to  the  granularity  specified  by  the  procuring  activity),  an  assignee,  the 
corrective  action,  system  components  modified,  and  effort  required  for  correction.  The  developer 
shall  provide  reports  of  discrepancies  and  corrective  actions  to  the  procurement  activity  in  accordance 
with  contract  requirements  or  development  agreements. 

Metrics  on  DRs  shall  be  collected  and  used  to  assess  test  program  progress  and  effectiveness, 
measure  trends,  provide  measures  of  defect  levels  and  manage  the  progress  of  discrepancy  resolution. 
The  specific  metrics,  collection  and  reporting  frequency,  and  analysis  required  shall  be  as  specified  in 
the  metrics  plan  approved  by  the  procuring  activity. 

5.8  Tests  During  Sustainment  Phase 

5.8.1  Hardware 

During  sustainment,  two  types  of  hardware  testing  are  required.  Maintenance  tests  shall  be  performed 
at  routine,  scheduled  maintenance  intervals  to  verify  that  hardware  performance  is  maintained.  In 
addition,  tests  shall  be  performed  to  demonstrate  that  performance  capabilities  are  maintained  after 
hardware  replacements  are  made.  For  both  types  of  sustainment  tests,  subsets  of  the  overall  system- 
level  testing  shall  be  performed  to  demonstrate  end-to-end  performance. 

Replacement  units  shall  undergo  the  same  testing  at  a  unit  and  subsystem  level  as  required  during 
qualification: 

1 .  Interface  requirements  for  the  replacement  items  shall  be  verified 

2.  End-to-end  subsystem  tests  such  as  bit  error  rate  (BER)  shall  be  performed 

3.  BITE  functionality,  if  present,  of  the  replacement  units  shall  be  demonstrated 

5.8.2  Software 

Software  tests  during  sustainment  shall  be  conducted  in  accordance  with  the  test  requirements  listed 
above  for  development.  Unit  tests  shall  be  conducted  for  all  units  with  modified  software.  Integration, 
subsystem  qualification,  regression  and  acceptance  tests  may  be  selected  subsets  of  previous  tests 
plus  new  tests  based  on  the  extent  of  the  required  modifications.  A  test  designed  to  demonstrate  that  a 
defect  has  been  corrected  must  be  demonstrated  to  fail  when  executed  on  the  software  without  the 
correction. 

5.8.3  System  Tests 

A  minimum  suite  of  integrated  system  tests  and  system-level  regression  tests  shall  be  established,  to 
be  performed  following  any  sustainment  modifications  to  the  ground  system.  In  addition,  each 
modification  during  sustainment  shall  be  assessed  to  determine  if  additional  unique  tests  are  to  be 
performed  to  validate  ground  system  readiness  for  return  to  operations.  A  test  designed  to 
demonstrate  that  a  system  failure  has  been  corrected  must  be  demonstrated  to  fail  when  executed  on 
the  system  without  the  correction. 
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6.  Notes 


This  section  contains  information  of  a  general  or  explanatory  nature  that  may  be  helpful,  but  is  not 


mandatory. 

6.1  Data  Item  Descriptions  (DIDs) 

Test-related  DIDs: 

Acceptance  Test  Plan 

DI-QCIC-80553 

Electromagnetic  Interference  Test  Procedures 

DI-EMCS-80201 

Electromagnetic  Interference  Test  Report 

DI-EMCS-80200 

Reliability  Test  Procedures 

DI-RELI-80251 

Reliability  Test  Reports 

DI-TMSS-81586 

Software  Test  Description 

DI-IPSC-81439 

Software  Test  Plan 

DI-IPSC-81438 

Software  Test  Report 

DI-IPSC-81440 

Test  Plan 

DI-NDTI-80566 

Test  Procedure 

DI-NDTI-80603 

Test/Inspection  Report 

DI-NDTI-80809 

Test  Plans  and  Procedures 

DI-SESS-81704 

Current  versions  of  these  DIDs  can  be  found  at  the  Acquisition  Streamlining  and  Standardization 
Information  System  (ASSIST)  website:  https://assist.daps.dla.mil/quicksearch/ 

6.2  Acronym  List 


BER 

Bit  error  rate 

BITE 

Built-in  test  equipment 

COTS 

Commercial-off-the-shelf 

CPU 

Central  processing  unit 

DID 

Data  item  description 

DOD 

Department  of  Defense 

DR 

Discrepancy  or  deficiency  report 

DT&E 

Development  test  and  evaluation 

EMC 

Electromagnetic  compatibility 

FAT 

Factory  acceptance  test 

FCA 

Functional  configuration  audit 

FCC 

Federal  Communications  Commission 

GFP 

Government-furnished  property 

IV&V 

Independent  verification  and  validation 

NDI 

Non-developmental  item 

OT&E 

Operational  test  and  evaluation 

PCA 

Physical  configuration  audit 

RF 

Radio  frequency 

RFR 

Run  for  record 

SAT 

Site  acceptance  test 

SDF 

Software  development  folder,  or  file 

31 


SOW 

Statement  of  work 

SVR 

System  verification  review 

TDF 

Test  development  folder 

THD 

Total  harmonic  distortion 

TRB 

Test  review  board 

TRR 

Test  readiness  review 

VCRM 

Verification  cross-reference  matrix 
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